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ABSTRACT 



This study evaluated the development of generic skills by 
students in the Arts Faculty at York University (Ontario, Canada) by 
comparing entering students and graduating students on a self-report measure. 
The specific skills evaluated included analytical, communication, personal, 
organizational, comparative, job procuring, basic numeracy, and computer 
skills. The study was based on a "value added" approach to measuring the 
effectiveness of the college experience. A questionnaire was developed and 
completed by 480 graduating students and 812 entering students in Fall of 
1995. Overall, the study found that graduating students scored higher than 
entering students on all measures of skill and the magnitude of change 
observed was comparable to that observed in similar studies conducted in the 
United States. Correlations among high school grades, cumulative grade point 
averages, and various skills, however, was low suggesting that good high 
school and university grades do not necessarily indicate high levels of 
generic skills. (DB) 
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Foreword 

The Institute for Social Research produces four types of articles in its publication series: 

• Working papers; 

• Reports on various technical and managerial aspects of the research process designed for 
technical support staff and research managers; 

• Reports on topics of general interest to non-specialist readers; and, 

• Reports on various methodological and substantive issues aimed at experts in the 
field. 

The following is a working paper. 
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While the development of generic skills is viewed by many academics and 
employers as a desirable outcome of the university experience, the extent to which 
this objective is realized is seldom measured. In this report differences in analytical, 
communication, personal, organizational, comparative, job procuring, basic 
numeracy, and computer skills between students entering, and graduating from, the 
Faculty of Arts at York University are measured through self-reports in surveys. 

Overall it is found that graduating students score higher than entering students on all 
measures of skill. Moreover, the magnitude of change is comparable to that 
observed in studies conducted in the United States. Correlations among high school 
marks, cumulative grade point averages, and various skills, however, are low. These 
findings suggest that while in Arts graduating students display higher generic skill 
levels than those entering the university, good high school and university marks do 
not necessarily indicate high levels of generic skills. Whether or not absolute skill 
levels of graduating students are sufficient is beyond the scope of this inquiry. 
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Introduction 



In 1995-96 the Vice President (Academic Affairs) of York University established 
two important task forces. The first focused on ways in which the first year 
experience could be changed so that students would be better prepared for the 
demands of university life. The second centred on the development of performance 
indicators that were related to York’s mission. 

Discussions in the first task force readily led to the realization that the specification 
of learning objectives logically preceded both an evaluation of the current first year 
experience and the formulation of policies intended to improve that experience. The 
learning objectives eventually accepted by the task force included the development 
of several generic skills, such as written and oral communication and critical skills, 
as well as the acquisition of subject matter expertise. While it was recognized that 
increases in students’ generic skills characteristically occur in disciplinary and inter- 
disciplinary contexts, it was nonetheless agreed that the development of generic 
skills was different from the acquisition of subject matter knowledge. 

The concerns of the first task force unintentionally dove-tailed with those of the 
second. Among other indices of performance, it was agreed that increases in 
students’ competencies between first and final years was an important aspect of 
institutional performance - some would say the most important measure of 
performance. In other words: does the York experience add value in terms of 
phenomena such as disciplinary expertise and generic skills? 

Abstracting from the deliberations it can be argued that both task forces had in 
common a concern with the specification of learning objectives, putting into practice 
policies designed to realize the objectives, and determining whether or not objectives 
had been met. To focus on the last of these concerns, while assessing increases in 
subject matter competence is not easy, measuring generic skills development is more 
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difficult still. For one thing, general concepts like analytical and critical skills mean 
different things to different people. Even if agreement can be reached at the 
conceptual level, the development of measures that operationalize the concepts is an 
additional hurdle. For another, while the disciplinary apparatus of the university is 
designed to evaluate students’ performance in subject matter areas, it is not set up to 
deal with assessments of generic skills. As a result, alternate means must be 
developed to measure improvements in these areas. Consistent with this objective, 
the current report will focus on a relatively cost effective attempt, through surveys, 
to measure generic skills of students entering, and graduating from, the Faculty of 
Arts, and to assess the value added to generic skills by the university experience. 

Although the Vice President (Academic Affairs) and the Dean of Arts supported the 
research on which the report is based, responsibility for the study’s design, analysis, 
and conclusions rests with the author. 

The Need for Generic Skills 

In recent years increasing attention has focused on the extent to which students in 
universities develop generic skills, such as communication skills, critical skills, and 
personal skills. Usually, however, universities neither attempt to systematically 
develop such skills nor do they try to measure the degree to which the university 
experience contributes to development in generic skills. Part of the explanation for 
this state of affairs may be that there is no consensus regarding the skills 
universities should develop and that operationalization and measurement of generic 
skills is difficult and costly. 

As evident in the pages of business publications such as Canadian Business and 
Fortune, in both Canada and the United States, business is putting increased 
emphasis on the need for the development of generic skills. For example, as well as 
subject matter expertise, the Conference Board of Canada argues that employers 
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want employees who can communicate effectively, think critically, have a 
commitment to continuous learning, demonstrate positive attitudes and behaviours, 
are responsible and adaptable, and can work well with others (Conference Board of 
Canada, 1992). 

While many may feel that university curricula should not be driven by the needs of 
business, many of the skills that business has identified as important, such as 
effective communication and critical thinking, are completely consistent with the 
objectives of a liberal education. Unfortunately, in many, if not most, universities, 
when generic skills are even discussed, it is assumed that they will develop as a 
natural by-product of studies in various disciplines. Moreover, few, if any, 
institutions identify the acquisition or honing of generic skills as desired outcomes 
of the university experience. 

The Value Added Approach 

When examining the extent to which students’ experiences in any given university 
contribute to the development of generic skills or any other desired outcome, it is 
important to obtain measures of the outcome under consideration at entry and again 
at graduation so that the ‘value added’ can be assessed. In essence, measures should 
be taken of the generic skills of students upon entry to the university and for the 
same students again upon graduation (Astin, 1991). For both entering and 
graduating students a utilization of skills in tasks that can be observed or in tests 
that can be evaluated is desirable (see Pascarella, Nora, and Bohr, n.d., for a 
discussion of the National Study of Student Learning in the United States that 
involves this type of assessment). Similar measures should be taken of a same-age 
control group that does not attend university. Increases in skills between entering 
and graduating students that are not also observed in the control group can be 
attributed to the university experience. 
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Design Problems 

In carrying out research of this nature, there are a number of practical matters that 
intercede between the ideal design and actual research. Primarily, longitudinal 
research that would measure students’ and a control group’s skills at appropriate 
intervals is very costly. Because many students might leave the university or refuse 
to participate in both the entry and exit measurement of skills, large samples would 
be required to ensure that at graduation sufficient numbers were still involved in the 
study to facilitate meaningful analysis. Even if a large sample remained at 
graduation the possibility exists that non-participants would be different from 
students who remained in the study. Control group attrition would also be a 
problem. 

A less costly cross-sectional design involves assessing the skills of one group of 
students at entry and another at graduation after controlling for variables having the 
potential to influence skills acquisition, such as previous levels of achievement (high 
school marks), gender, language spoken in the home, and racial origin. If 
comparisons were to show that graduating students had skills not evident among 
entering students and a same-age control group differences between the entering and 
graduating students could be attributed to the university experience. Variations of 
this method have been used successfully in other studies (Keeley, Browne and 
Kreutzer, 1982; Steele, 1986). 

One problem with this approach is that when they entered university, the graduating 
students may have been different from the entering group with whom they are being 
compared. For example, if in recent years an increased emphasis had been placed 
on generic skills in high schools, entering students would demonstrate higher levels 
of generic skills than would have been displayed by graduating students when they 



§ 



Value Added in Generic Skills Between First And Final Year 



O 

ERIC 



were in first year. 1 

Although both longitudinal and cross-sectional studies require an external control 
group if the possible effects of the university experience are to be disentangled from 
those of maturation, control groups are frequently absent from studies of university 
outcomes. While this presents difficulties if the intent is to assess the effect of the 
university experience per se, provided same-age groups are under consideration, it is 
less problematic if the research objective is one of assessing the impact of different 
institutional contexts on outcomes. For example, if the outcomes of one university, 
faculty, or department are being compared with those of other universities, faculties, 
or departments, provided that same-age groups are involved in the study and that 
pre-entry characteristics are held constant, an external control group is not required. 



Self-Reports 



Independent of whether the research is longitudinal or cross-sectional, measuring 
skills through standardized tests can be an expensive proposition. For example, 
leaving aside the costs of fees for using standardized tests, instrument scoring, test 
administrators, and communications with students, participants in the study of 
outcomes currently being undertaken by the National Centre on Postsecondary 
Teaching, Learning, and Assessment in the United States are being paid $30.00 
(U.S) for taking the first round of tests and the stipend increases by $5.00 each time 
additional tests are taken. A less costly way of measuring skills involves self- 
reports in surveys. 

While self-reports may be viewed as less desirable than more ‘objective’ measures 
of, in this instance, skills, as Pike ( 1995 a: 1) points out, in the United States the 



’Despite the desirability of longitudinal research in studies of this nature, Pascarella and 
Terenzini ( 1 99 1 : 1 24) point out that in studies carried out on reflective judgement, 
longitudinal and cross-sectional research produce similar results. 
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National Education Goals Panel Resources Group on Adult Literacy and Lifelong 
Learning noted that the development of assessment tools of use at the national level 
to measure university outcomes would cost several million dollars and take many 
years to complete. In the interim the Group recommended that self-reports be used 
as proxies. 

To what extent can self-reports be viewed as reasonable proxies of skills, and/or 
attitudes and/or knowledge? After examining a number of studies in which the 
results of self-reports were compared to test results, Pascarella and Terenzini 
(1991:1 00) conclude that correlations between self-reports and other measures of 
the same phenomena range from .25 to .65. More importantly, the literature they 
draw upon in their examination indicates that the validity of self-reports varies with 
the specific skills, and/or attitudes and/or knowledge under discussion. For 
example, Berdie (1971) found a high correlation between self-professed knowledge 
of public figures and the results of tests designed to measure knowledge of the same 
figures; however, the relationships between self-reported and tested knowledge of 
authors and artists were not as high. Similarly, Pohlmann and Beggs (1974) 
discovered that self-reports of academic growth in the affective realm were 
supported by test results. Growth in simple and complex cognitive realms, 
however, did not correlate highly with self-reports. In a review of the literature on 
the utility of self-reports Baird (1976) cites examples of both high and low 
relationships with external measures of various phenomena. Still others have 
presented information suggesting that while self-reports have some uses, they 
should not be viewed as substitutes for other measures of various college or 
university outcomes (McMorris and Ambrosino, 1973; Dumont and Troelstrup, 
1980). 

In view of the objective of the current undertaking, perhaps the most important 
research to have been undertaken on the relationship between self-reports and other 
measures of knowledge is reported by Pike (1995a). In brief, Pike studied 1,568 
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graduating students from 10 colleges and universities in the United States who 
completed the College Basic Academic Subjects Examination (College BASE) 
adapted for graduating students. The College BASE tests proficiency in English, 
mathematics, science, and social science. Students who completed the examination 
also participated in a survey in which they rated their ability on exactly the same 
matters covered in the examination. The correlations between test performance and 
self-reports were sufficiently high for Pike to conclude that self-reports can be used 
as general proxies for traditional measures of academic achievement. He 
emphasizes, however, that there must be a high content correspondence between 
self-report questions and those asked in tests. While this may seem self evident, 
other research has focused on the relationship between some general self-report 
measures and the results of specific tests. 

Although this and other work by Pike (1994, 1995b) is encouraging, it should be 
stressed that the focus of the research has been on knowledge (not generic skills) as 
measured through the College BASE. Although the current research assumes a 
similar relationship between self-reported skills and the results of tests designed to 
measure such skills, research is necessary to confirm the connection. 

The Important Skills 

Because there is no consensus in Canada or the United States regarding the generic 
skills that are both important for university graduates to possess and that should be 
developed throughout a student’s university career, in this study, a list of desirable 
skills was developed through an inductive process. In an iterative process, faculty 
members with knowledge of skills development, along with researchers from the 
Institute for Social Research at York University, identified a number of tasks, the 
performance of which ideally would be improved over the course of a university 
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Table 1: Skill Categories and Question Topics 



Analytical Skills 

• identifying the main points in lectures in your major 

• clearly identifying the pros and cons of controversial issues like abortion 

• figuring out the main arguments in articles written on topics in your discipline 

• identifying flaws in positions given by other students in classes or seminars 

• explaining your strengths and weaknesses to a potential employer 

• defending a position you have taken in a classroom or seminar against the criticisms of other 
students 

Communication Skills 

• taking an article you read for a course this year and summarizing it in no more than two pages 

• verbally presenting your ideas on a topic of your choice to a group of ten strangers 

• verbally communicating to other students in your classes the flaws in their positions or arguments 

• writing a letter to a friend 

• writing a letter of application for a job 

• expressing yourself clearly in written English in an essay 

• correcting the grammar and spelling in the essay of a friend 
Personal Skills 

• assessing the feelings of people you have worked with in part-time or summer jobs 

• 'cooling out' a friend who is annoyed with you 

• being able to apologize to someone if you said something wrong about them 

• being able to admit to yourself when you are wrong 

• being able to get along with other people 

• enjoying meeting new people 

• helping friends patch up disagreements 

• considering the feelings of others before doing things 

• knowing yourself 
Organizational Skills 

• planning a job search strategy for a friend 

• organizing priorities to prevent rushing at the last minute 

• recruiting and organizing twenty strangers to collect money for the Heart and Stroke Fund 
Comparative Skills 

• comparing what is going on in Canada to what is going on in a third world country 

• comparing what is going on in Canada today to what was going on fifty years ago 

• comparing what is going on in Canada to what is going on in any European country 

Job Procuring Skills 

• knocked on neighbors doors for odd jobs 

• phoned companies about advertised job 

• sent resumes to companies that might be hiring 

• phoned companies that might be hiring 

• gone to companies to see if jobs are available 

• started own business (painting, catering, etc.) 

Basic Numeracy Skills 

• determining change from a $10 bill for a $2.75 pen 

• calculating 15% discount on a $9.36 book 

• determining percentage Liberal vote when 15 students vote Conservative, 10 NDP, and 20 
Liberal 

• solving for 'x' in the equation 3x-5=56.7 

• explaining the meaning of 'square root' 

Computer Skills 

• using a word processing program 

• using a spread sheet program 

• using a statistical analysis program 
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career. 2 (The exception is job procuring skills for which students were simply asked 
if they had engaged in particular activities.) Next, questions were developed that 
focused on the difficulty students would have in completing specific tasks. Finally, 
questions were grouped into logical categories. In keeping with Pike’s finding 
noted earlier, tasks, and their related questions, were very specific and relevant to 
students ’ experiences. 

Survey questions were tested in a small pilot survey involving students about to 
graduate from Pure and Applied Science, Arts, Fine Arts, and Administrative 
Studies. For each question, excluding those on job procuring skills for which 
students were asked how often they completed particular activities, on a five point 
scale, responses of 1 indicated a high, and 5 a low, level of difficulty in completing 
the specific task referenced in the question. On the basis of the pilot, some 
questions were dropped and assessments were made of the reliability of indices 
comprised of various questions. The relevant skill categories and question topics 
used in the construction of indices are displayed in Table I. 

While performance on the tasks in each of these skill categories may improve over 
the student’s university career, it cannot automatically be assumed that change is a 
result of the university experience. Improvements in tasks included in the analytical, 
communication, and comparative skills categories may result primarily from the 
challenges of university life (both curricular and non-curricular); however, it is 
likely that maturation has a role to play in the development of personal skills and 
perhaps also in organizational skills. Improvements in job procuring activities may 
reflect economic necessity as much as anything else. Independent of their origins, 
the important point is that such skills may be essential to a successful post- 
university career and should be part of the skills repertoire of graduating students. 



2 Ron Sheese, Nick Elson, Darla Rhyne, Tammy Chi, David Northrup, and Paul Grayson 
participated in this phase of the study. 
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The Sample 

Information for the analysis of value added in generic skills between first year and 
graduation was collected in two surveys conducted in the Fall of 1995. The first 
included all students eligible to graduate in the Fall convocation. In total 480 Arts 
students returned completed questionnaires as did 25, 30, and 6 students from the 
Faculties of Fine Arts, Science, and Administrative Studies respectively. Low 
numbers reflect the facts that compared to Arts other faculties are small and 
relatively few students graduate in the Fall. Larger surveys of Spring graduates 
' carried out in 1996 will facilitate analyses of faculties other than Arts. The response 

rate for the survey was 58%. 

At the same time as the survey of graduates, questionnaires assessing generic skills 
were mailed to all students entering the faculties of Fine Arts, Science, and 
Administrative Studies and to a sample of students entering the Faculty of Arts. 

The numbers of returns were 171, 223, 100, and 812 respectively. The response 
rate, approximately 55%, was comparable to the study of graduating students. For 
both studies, survey data were merged with information obtained from 
administrative files on marks. 

Unfortunately, as with much of the research focusing on value added by the 
university experience, resources did not permit the establishment of an external 
control group. Even if resources had not been an issue, given how some of the 
specific skill constellations under discussion were operationalized, it is doubtful that 
a comparison with an external control group would have been meaningful. For 
example, being able to identify the main points in lectures in your major is irrelevant 
to someone with no post-secondary experience. Comparisons with ‘mature’ 
students would have been more germane. Were it to be found that no differences 
existed between students in their late teens and older students at a similar study level 
it could be argued that differences in generic skills between entering and graduating 
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students were not a result of maturation. Such comparisons will be possible after 
the collection of information from departmental reviews at Atkinson College, the 
evening operation of York University, that is attended by students older than those 
in Arts. 

Reliability of Indices 

The reliability of the various indices of skills as outlined in Table 1 was tested in the 
surveys of graduating and entering students by using Cronbach’s alpha which “can 
be viewed as the correlation between this test or scale and all other possible tests or 
scales containing the same number of items, which could be constructed from a 
hypothetical universe of items that measure the characteristic of interest” (Norusis, 
1992: 149). In general, indices having alphas at or above .7 can be regarded as 
reliable. 

Cronbach’s alpha for each of the skill indices used in this analysis, stratified by 
survey (entering and graduating), are provided for each faculty in Table 2. For both 
the entering and graduating samples, with the exception of organizational skills, 
overall alphas are acceptable for each index. Although overall alphas are acceptable 
for job procuring and basic numeracy skills in the entering survey, alphas for job 
procuring skills in Administrative Studies, and basic numeracy skills in both Science 
and Administrative Studies, are slightly lower than .70. Similarly, in the graduating 
sample, alpha for basic numeracy skills for students in Science is considerably less 
than .70. As the current analysis focuses on students in Arts, these deviations are of 
no current consequence. In essence, in the current analysis, with the exception of 
organizational skills, all overall alphas, and alphas for each skill within faculties, are 
acceptable. 3 

3 A principal components analysis using all of the skills variables was conducted in 
which the number of factors specified for retention, 7, was equal to the number of indices 
that had been constructed. With factor loadings lower than .4 suppressed, the analysis 
yielded factors similar to those of basic numeracy, computer skills, personal skills. 
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Table 2: Cronbach's Alpha for Skills Indexes 
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Correlations Among Skills and Grades 

Correlations among each of the skills and Ontario Academic Credit (OAC) marks 
are detailed in Table 3 (maximum cases 804, minimum 729) and Table 4 (maximum 
cases 477, minimum 328). The first thing of note is that for students entering Arts 
correlations between OAC marks and various skills are low. Of those that are 
statistically significant, there is a positive correlation between communication skills 
and OACs (.10); however, for personal and job procuring skills the correlations are 
negative (-.11 and -.10 respectively). 

There are two ways in which this general finding can be interpreted. First, if one 
expects that generic skills are a necessary condition for high school achievement as 
measured in OAC marks, the low correlations can be taken as in indication that the 
current measures of generic skills are invalid: students with high OAC marks must 
possess high generic skills. On the other hand, if it can be accepted that OAC 
marks do not necessarily reflect generic skills the findings make sense. Certainly 
many faculty who bemoan the lack of preparation on the part of first year students 
despite high OAC marks would be amenable to this interpretation. At the same 
time, if employers are to be believed, it is equally likely that there is little 
relationship between high university grades and generic skills (Jones, 1 994). 

Unfortunately, support for employers’ concerns is found in Table 4. For graduating 
students there are statistically significant, but low, correlations (. 14 and .16 
respectively) between analytical and communication skills and cumulative grade 
point average (GPA). The statistically significant correlation between job procuring 
skills and GPA is negative and small (-.09). No other correlations are statistically 
significant. In essence, there is at best a weak relationship between some generic 



comparative skills, and job procuring skills. The other factors combined, in no systematic 
fashion, variables that comprised the analytical and communication skill categories (see 
Spector (1992:54) for a rationale). On the basis of these findings it was deemed appropriate 
to retain the original indices. 
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skills and GPA: marks may measure subject knowledge but not generic skills. 

Unfortunately, once again, we are left with a quandary. If we assume that university 
graduates must possess the generic skills analysed here, then we must also conclude 
that the skills measures employed in the study are invalid. If, on the other hand, we 
accept the views of critics that university curricula do not instill generic skills the 
findings help validate the skills measures employed in the study. 

While the link between generic skills and GPA may be weak, the correlation 
between cumulative GPA and OAC marks is a statistically significant .55. This 
relationship is comparable to the findings of studies of graduating students in the 
United States (Astin, 1993: 188) and to the relationship between OAC and first year 
marks for Arts students at York (Grayson, 1995: 103). In essence, OAC marks 
maintain a reasonably strong relationship with academic achievement over the 
students’ university careers. 

Control Variables 

In order to determine value added, it is necessary to compare the scores on various 
skill indices of graduating to entering students. While standardized scores are not 
necessary to achieve this objective, there were two reasons for calculating them in 
the current undertaking. (For most practical purposes standardized scores (or z- 
scores) have a mean of 0 and a standard deviation of 1 .) First, in order to compare 
changes at York to changes reported in the literature, and to make changes 
understandable to the average reader, it is necessary to specify differences between 
graduating and entering students in terms of percentile point differences. Second, 
standardized rather than actual scores will be reported in order to protect the 
confidentiality of the York data. In essence, while readers will know the amount of 
change that can be attributed to the university experience, they will not know the 
absolute value of entering and graduating scores (however, such data are available 
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to the York community). From a value added perspective the former information is 
more important than the latter: the concern is not with the level of generic skills of 
graduating students but with how much the university experience has contributed to 
the development of generic skills. 

In the analysis of net gains in various skills, it is important to control for possible 
confounding variables such as gender, racial origin, and language spoken in the 
home while the student was growing up. That each may have implications for 
particular skills is evident from Table 5 that includes standardized skills scores for 
entering and graduating students combined. 

Unfortunately, space constraints prevent a detailed examination of data in Table 5. 

At a very general level, however, it is obvious that considerable differences in skills 
are based on gender, racial origin, and home language. For example, males score 
highest on basic numeracy and computer skills. Females do better than males on 
personal and organizational skills. There are no statistically significant differences 
for analytical, comparative, and job procuring skills. 

If racial origin is examined. Black students score highest on analytical, 
communication, and job procuring skills. Students of European origin have the 
highest scores on personal and comparative skills and they tie with Black students 
for high scores on organizational skills. The highest scores on basic numeracy skills 
are reported by students of Chinese origin. Differences in computer skills are not 
statistically significant. 

One cause for alarm is the low rating for students of Chinese origin on analytical, 
communication, and comparative skills. As the majority of these students did not 
speak English in their homes, these figures likely illustrate the impact of language 
on various skills areas. In a relatively language neutral area, like basic numeracy, 
students of Chinese origin score high. 
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The importance of language is further illustrated when language spoken in the home 
is examined. For all skills, with the exception of basic numeracy skills, students 
who spoke English in the home score higher than those who used other languages. 
Computer skills was the only area in which differences based on home language 
were not statistically significant. 

Assessing Value Added 

The percentile point difference in the skills of graduating as compared to entering 
Arts students, as determined through analyses of covariance in which OAC marks, 
gender, race, and language spoken in the home were controlled, is summarized in 
Graph 1 . 4 Overall, the greatest net gains were for analytical and communication 
skills (24 percentile point gains for each). Organizational, basic numeracy, 
personal, and comparative skills increased by 20, 14, 12, and 1 1 percentile points 
each. Lowest increases, 9 percentile points in each case, were found for job 
procuring and computer skills. 

An Overall Assessment 

How can we evaluate the differences in generic skill scores of students entering and 
leaving the Faculty of Arts at York University? Fortunately, in their tome, How 
College Affects Students, Pascarella and Terenzini ( 1 99 1 :558), in Table 6, have 
summarized findings from a number of studies on freshman to senior gains in a 
number of domains similar to those analysed here. 

The first column of the table lists the skills under consideration. Column two lists 
the ‘effect size’ that can be viewed as the difference between the mean score for 




4 Despite differences in Table 5, there was little difference in scores with and without 
controls. Similarly, scores differed little when the number of entering students were matched 
to the number of graduating students. 
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Graph 1: Value Added in Generic Skills Between First and Final Year In Faculty of Art 

in Percentile Points 





Table 6: Summary of Estimated Freshman-to-Senior Changes: Learning and Cognitive 
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graduating and entering students divided by the entering student standard deviation. 
(This value is equivalent to the unadjusted mean in the analysis of covariance 
conducted for this report.) Column three contains information of effect size 
translated into area under the normal curve. 

There are two important observations that can be made on the basis of the data in 
Table 6. First, overall gains between first and final year are modest ranging from a 
low of 10 percentile points for quantitative skills to a high of 38 percentile points 
for ability to deal with conceptual complexity. Second, while different measurement 
techniques preclude direct comparisons, there is a great deal of similarity between 
differences observed at York and those seen in studies conducted elsewhere. 

For example, Pascarella and Terenzini report gains of 21 percentile points for 
general verbal skills, 22 points in oral communication skills, and 19 percentile 
points in written communication skills (the average for these three is 2 1 percentile 
points). At York, gains in communication skills, that included some general verbal 
as well as written and oral components, were 24 percentile points. The same is true 
of quantitative skills: Pascarella and Terenzini report gains of 10 percentile points 
and the gain in the Faculty of Arts was 14 points. In the realms of Piagetian 
reasoning, critical thinking, use of reason, and ability to work with conceptual 
complexity, Pascarella and Terenzini record average entering to graduating 
differences of 30 percentile points ((13+34+34+38)/4). In Arts the gain for 
analytical skills, that includes some of the components of the foregoing, was 24 
percentile points. 

It must be stressed that the studies summarized by Pascarella and Terenzini used 
many different measures of the skills under consideration (some involved the use of 
standardized tests of various skills). As a result, it would be wrong to place too 
much emphasis on specific comparisons between their and the York findings. What 
is important is the fact that gains observed at York are similar in size to those 
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measured elsewhere; therefore, it can be argued that in terms of generic skills, the 
value added of an Arts education at York may in general be comparable to the value 
added in many U.S. universities. More importantly, the use of surveys at York 
produced gains similar to those elsewhere that used a variety of measurement 
techniques. 



Broader Implications 

Students graduating from the Faculty of Arts at York University report higher 
generic skills than entering students. Scheduled research will determine if similar 
outcomes can be found in the faculties of Fine Arts, Pure and Applied Science, and 
Administrative Studies. In addition, on-going program reviews, that will include the 
questions on generic skills discussed in this report, will facilitate similar 
examinations at the departmental level. Unfortunately, the degree to which value 
added in generic skills at York is comparable to that in other Canadian universities 
is contingent upon the latter opening comparable avenues of inquiry. 



In the event that similar research were initiated, what might be expected? Research 
carried out in the United States can be of assistance in answering this question. As 
Pascarella and Terenzini (1991:589), after their review of approximately 3,000 U.S. 
studies related to university outcomes, argue: 

There are clear and unmistakable differences among postsecondary 
institutions in a wide variety of areas, including size and complexity, 
control, mission, financial and educational resources, the scholarly 
productivity of faculty, reputation and prestige, and the characteristics of 
the students enrolled. At the same time, however, American colleges and 
universities also resemble one another in a number of important respects. It 
may be that despite their structural and organization differences, their 
similarities in curricular content, structures, and sequencing; instructional 
practices; overall educational goals; faculty values; out-of-class 
experiences; and other areas do in fact produce essentially similar effects 
on students although the ‘start’ and ‘end’ points may be very different 
across institutions. 
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Given that U.S. universities are far more diverse than those in Canada, it is likely 
that in this country there is even a smaller difference in outcomes such as those 
studied here than south of the border. In other words, while institutions like York, 
Memorial, Queen’s, Ryerson Polytechnic University, the University of Toronto, 
Western, the University of Manitoba, the University of Regina, and the University 
of Victoria may vary in things like resources and the entering average of students, 
based on findings from U.S. research, it is unlikely that great differences exist in the 
value added to generic and other skills. 



Discussion 

As a minimum, data collected in this study can be viewed as students’ perceptions of 
their standing on measures of generic skills. Although some other studies have 
found reasonable correlations between self-reports as used here and other skills 
indicators, a conservative approach requires further validation of the measures used 
in this study before it will be possible to go beyond the level of perception. This 
does not imply that students in general are unaware of their strengths and 
weaknesses: the absolute ratings on many of the skills questions (that were not 
shown) indicate the contrary. 

Although we will not know until comparisons are made between self-assessed skills 
of traditional and mature students at similar study levels the degree to which 
differences reported by entering and graduating students are a function of 
maturation, given the ways in which many skills tasks are operationalized, it is 
doubtful that similar competencies would have been developed by a same-age 
control group that did not attend university. Moreover, for some important 
purposes, the maturation factor is irrelevant. For example, if the concern is with the 
degree to which students at a particular study level in department x, faculty y, or 
university z believe that they can identify the pros and cons of controversial issues 
like abortion, or spot the flaws in arguments of other students, maturation is not an 
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issue: the students being compared are of a similar age. As a result, in terms of 
policy, a finding, for example, that sociology students rate their analytical skills 
lower than those in psychology might lead the sociology faculty to examine their 
program closely to see if they can introduce changes that would increase the 
development of analytical skills. As a result, self-reported data on generic skills can 
assist in the process of departmental and program reviews as well as provide an 
indication of skills differences between entering and graduating students. 

Conclusions 

The first conclusion that derives from the foregoing analysis is that at York 
University students who graduate from the Faculty of Arts leave the university with 
higher levels of generic skills than students who enter the faculty. Despite this 
increase in generic skills, cumulative GPAs vary little by skill levels. This finding 
suggests that generic skills as measured here may not be essential to the 
achievement of good grades and good grades do not indicate high levels of generic 
skills. 

A second conclusion from the analysis is that the magnitude of the gains in generic 
skills is comparable to those observed in studies of outcomes of American 
universities and colleges. Whether absolute skill levels, that can be assessed by 
reference to unstandardized scores on skills indices, are acceptable, is beyond the 
scope of this inquiry. 
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